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Multimodal, speech-enabled systems pose different research problems when compared to 
unimodal, voice-only dialogue systems. One of the important issues is the question of how 
a multimodal interface should look like in order to make the multimodal interaction 
natural and smooth, while keeping it manageable from the system perspective. Another 
central issue concerns algorithms for multimodal dialogue management. This paper 
presents a solution that relies on adapting an existing unimodal, vocal dia ... 
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This paper addresses two related topics: Firstly, it presents building-blocks for flexible 
multimodal dialog interfaces based on standardized components (VoiceXML, XML) to 
indicate that thanks to well-supported standardizations, mobile multimodal interfaces to 
heterogeneous data sources are becoming ready for mass-market deployment, provided 
that adequate modularization is respected. Secondly, this is put in the perspective of a 
discussion of knowledge management in firms, and the paper argues t ... 
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Full text available: ^ pdf(300.86 KB) Additional Information: full citation , abstract , references , index terms 

That multimodal interfaces have benefits over unimodal ones has often been asserted. 
Several such benefits have been described informally, but, to date, few have actually • 
been formalized or quantified. In this paper, the hypothesized benefits of semantically 
redundant multimodal input actions are described formally and are quantified using the 
formalisms provided by Information Theory. A reinterpretation of Keates and Robinson's 
empirical data (1998) shows that their criticism of multimod ... 

Keywords: augmentative and alternative communication (AAC), interface evaluation, 
interventions for communication disorders, multimodal interfaces, speech generating 
devices (SGD), voice output communication aids (VOCA) 
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For computational Augmentative and Alternative Communication (AAC) aids, it has often 
been asserted that multimodal interfaces have benefits over unimodal ones. Several such 
benefits have been described informally, but, to date, few have actually been formalized 
or quantified. In this paper, some of the special considerations of this application domain 
are described. Next, the hypothesized benefits of semantically nonredundant multimodal 
input actions over unimodal input actions are descr ... 

Keywords: augmentative and alternative communication (AAC), interventions for 
communication disorders, multimodal interfaces, speech generating devices (SGD), voice 
output communication aids (VOCA) 
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Full text available: ^ pdf(507.59 KB) Additional Information: full citation , abstract , references , index terms ■ 

This article presents a User Interface (UI) framework for multimodal interactions targeted 
at immersive virtual environments. Its configurable input and gesture processing 
components provide an advanced behavior graph capable of routing continuous data 
streams asynchronously. The framework introduces a Knowledge Representation Layer 
which augments objects of the simulated environment with Semantic Entities as a central 
object model that bridges and interfaces Virtual Reality (VR) and Artificial ... 

Keywords: gesture and speech processing, multimodal interaction, semantic scene 
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An important aspect of the interpretation of multimodal messages is the ability to identify 
when the same object in the world is the referent of symbols in different modalities. To 
understand the caption of a picture, for instance, one needs to identify the graphical 
symbols that are referred to by names and pronouns in the natural language text. One 
way to think of this problem is in terms of the notion of anaphora; however, unlike 
linguistic anaphoric inference, in which antecedents for pronou ... 

7 Posters: Modelin g multimodal inte g ration patterns and performance in seniors: 
toward adaptive processin g of individual differences 
Benfang Xiao, Rebecca Lunsford, Rachel Coulston, Matt Wesson, Sharon Oviatt 
November 2003 Proceedings of the 5th international conference on Multimodal 

interfaces 
Publisher: ACM Press 

Full text available* 151 pdf(336 43 KB) Addit ' onal Information: full citation , abstract , references , citings, index 
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Multimodal interfaces are designed with a focus on flexibility, although very few currently 
are capable of adapting to major sources of user, task, or environmental variation. The 
development of adaptive multimodal processing techniques will require empirical guidance 
from quantitative modeling on key aspects of individual differences, especially as users 
engage in different types of tasks in different usage contexts. In the present study, data 
were collected from fifteen 66- to 86-year-old heal ... 

Keywords: human performance errors, multimodal integration, self-regulatory language, 
senior users, speech and pen input, task difficulty 
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Interaction designers are increasingly faced with the challenge of creating interfaces that 
incorporate multiple input modalities, such as pen and speech, and span multiple devices. 
Few early stage prototyping tools allow non-programmers to prototype these interfaces.. 
Here we describe CrossWeaver, a tool for informally prototyping multimodal, multidevice 
user interfaces. This tool embodies the informal prototyping paradigm, leaving design 
representations in an informal, sketched form, and create ... 

Keywords: informal prototyping, mobile interface design, multidevice, multimodal, pen 
and speech input, sketching 
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As a new generation of multimodal systems begins to emerge, one dominant theme will 
be the integration and synchronization requirements for combining modalities into robust 
whole systems. In the present research, quantitative modeling is presented on the 
. organization of users' speech and pen multimodal integration patterns. In particular, the 
potential malleability of users' multimodal integration patterns is explored, as well as 
variation in these patterns during system error handling and tasks ... 

Keywords: Gestalt theory, co-timing, entrenchment, error handling, multimodal 
integration, speech and pen input, task difficulty 
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We describe an approach to 3D multimodal interaction in immersive augmented and 
virtual reality environments that accounts for the uncertain nature of the information 
sources. The resulting multimodal system fuses symbolic and statistical information from 
a set of 3D gesture, spoken language, and referential agents. The referential agents 
employ visible or invisible volumes that can be attached to 3D trackers in the 
environment, and which use a time-stamped history of the objects that intersect ... 

Keywords: augmented/virtual reality, evaluation, multimodal interaction 
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This paper examines user behavior during multimodal human-computer interaction (HCI). 
It discusses how pointing, natural language, and graphical layout should be integrated to 
enhance the usability of multimodal systems. Two experiments were run to study 
simulated systems capable of understanding written natural language and mouse- 
supported pointing gestures. Results allowed to: (a) develop a taxonomy of 
communication acts aimed at identifying targets; (b) determine the conditions under 
which sp ... 

Keywords: cross-modal integration, referent identification strategies 
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This paper presents a multimodal interface architecture that combines standardized voice 
and ink formats to facilitate the creation of robust and efficient multimodal mobile e- 
Commerce systems, particularly for noisy mobile environments. The platform provides a 
Web interactive system for generic multimodal application development. By providing 
mutual disambiguation of input signals and superior error handling this architecture 
should broaden the spectrum of users to the general population, inclu ... 
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Although commercial dictation systems and speech-enabled telephone voice user 
interfaces have become readily available, speech recognition errors remain a serious 
problem in the design and implementation of speech user interfaces. Previous work 
hypothesized that switching modality could speed up interactive correction of recognition 
errors. This article presents multimodal error correction methods that allow the user to 
correct recognition errors efficiently without keyboard input. Correcti ... 

Keywords: dictation systems, interactive error correction, multimodal interfaces, pen 
input, performance model, speech input, speech user interfaces 
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Our research addresses the problem of error correction in speech user interfaces. Previous 
work hypothesized that switching modality could speed up interactive correction of 
recognition errors (so-called multimodal error correction). We present a user study that 
compares, on a dictation task, multimodal error correction with conventional interactive 
correction, such as speaking again, choosing Tom a list, and keyboard input. Results show 
that multimodal correction is faster than conve ... 

Keywords: interactive error correction, multimodal interaction, quantitive performance 
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Zhen, Wei 
Wang, Philip R. 

RE: 10/617422 - help with transfer 



Not suitable for 704 or 704/255. Claims 1 is dealing with interaction between software applications, there is no processing 
of signal. I believe claim 1 should be classified somewhere in 717 or 703/23. Claim 3 recites that the peripheral devices 
include speech synthesizer or speech recognition but they are just that peripheral devices and is insufficient to make it a 
subject matter for class 704, the same way claim 4 recites a screen does not make it the subject matter for class 345. If I 
missed something, let me know or send the examiner over to see one of the primaries or myself. Thanks -- 
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Richemond, 

Could you please reconsider this case for restriction requirement? Claims 23-33 will be 
classified under 717, Could 1-22 and 34 be classified in 704/255 (this is very similar to 
1 0/61 7,593 which is currently under your docket)? Thank you for your help. 

Wei Zhen 
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Claim 23-33 are directed towards "developing.. .applications. ..", but 1-22 & 34 are for executing speech 
processing. 

Seems like same examiner should do them all, and restrict the "developing. ..application claims" 
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